Skip to main content

Guide for setting up GCP

This guide helps you set up your Google Cloud Platform (GCP) project to run with DataStori.

DataStori uses Google Cloud Run Jobs to run the pipeline code. DataStori integrates with your GCP project using IAM Service Accounts.

Requirements​

Please be ready with the following resources:

VPC/Networking requirements​

  • VPC Network Name: The VPC where you want to run your code.
  • Subnet Name: The subnet where you want to run your code.
  • VPC Firewall Rules: Ensure firewall rules allow necessary egress. You can control access via network tags.

Service Requirements​

  • Cloud Run: DataStori will spin up Cloud Run Jobs to run the pipeline and store the output in Google Cloud Storage. Please ensure the Cloud Run API is enabled in your project.
  • Google Cloud Storage (GCS): Data is stored here. Please be ready with the Bucket Name where you want the data to be stored.
  • RDBMS (optional): If you want to push data to any other RDBMS.

IAM / Service Accounts​

We will create two service accounts:

  • One for the Cloud Run Job to use (workload identity).
  • One for DataStori to impersonate to manage the jobs.
  1. Create a Service Account for the Job: This service account will be used by the Cloud Run Job itself to access GCS.
    • Navigate to IAM & Admin -> Service Accounts -> Create Service Account.
    • Name it datastori-job-runner-sa.
    • Grant this service account the Storage Object Admin (roles/storage.objectAdmin) role so it can write to your GCS bucket.
    • Click Done. Note down the email of this service account.

  1. Create a Service Account for DataStori: This is the service account that DataStori's infrastructure will authenticate as.

    • Create another service account named datastori-integration-sa.
    • Do not grant this service account any roles directly in this step. Click Done.
    • Note down the email of this service account.

  2. Create a Custom Role: This role will contain the specific permissions DataStori needs to manage Cloud Run Jobs and use the job runner service account.

    • Navigate to IAM & Admin -> Roles -> Create Role.
    • Give it a title like "DataStori Job Manager".
    • Add the following permissions:
      • run.jobs.run
      • run.jobs.get
      • run.jobs.list
      • run.executions.get
      • run.executions.list
      • run.executions.delete
      • iam.serviceAccounts.actAs (Allows passing the runner SA to the job)
    • Click Create.

  3. Bind Permissions:

    • On the Project Level: Go to IAM & Admin -> IAM. Grant the datastori-integration-sa the custom "DataStori Job Manager" role you just created.
    • On the Job Runner Service Account: Navigate to the datastori-job-runner-sa you created in step 1. Go to the Permissions tab. Grant the datastori-integration-sa the Service Account User (roles/iam.serviceAccountUser) role. This allows the integration account to "act as" the runner account.
    • Allow DataStori to Impersonate: Navigate to the datastori-integration-sa. Go to the Permissions tab. Grant the DataStori main service account principal the Service Account Token Creator (roles/iam.serviceAccountTokenCreator) role. Please ask customer support for DataStori's principal service account email.

Logging (Optional)​

By default, DataStori will write the pipeline logs to Google Cloud Logging. If you want to customize the logging destination, please share details of your log sink configuration.

Summary​

To proceed with the GCP setup, please provide the following:

  1. GCP Project ID
  2. VPC Network Name
  3. Subnet Name
  4. GCS Bucket Name
  5. GCS Bucket Region
  6. The email of the datastori-integration-sa (Service Account for DataStori)
  7. The email of the datastori-job-runner-sa (Service Account for the job)
  8. Cloud Logging sink details (optional)